
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
  
<!-- Mirrored from werkzeug.palletsprojects.com/en/1.0.x/unicode/ by HTTrack Website Copier/3.x [XR&CO'2014], Tue, 15 Sep 2020 06:37:09 GMT -->
<head>
    <meta http-equiv="X-UA-Compatible" content="IE=Edge" />
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Unicode &#8212; Werkzeug Documentation (1.0.x)</title>
    <link rel="stylesheet" href="../_static/werkzeug.css" type="text/css" />
    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
    <link rel="stylesheet" type="text/css" href="../../../../assets.readthedocs.org/static/css/badge_only.css" />
    <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
    <script type="text/javascript" src="../_static/jquery.js"></script>
    <script type="text/javascript" src="../_static/underscore.js"></script>
    <script type="text/javascript" src="../_static/doctools.js"></script>
    <script type="text/javascript" src="../_static/language_data.js"></script>
    <script async="async" type="text/javascript" src="../../../../assets.readthedocs.org/static/javascript/readthedocs-doc-embed.js"></script>
    <link rel="shortcut icon" href="../_static/favicon.ico"/>
    <link rel="index" title="Index" href="../genindex/index.html" />
    <link rel="search" title="Search" href="../search/index.html" />
    <link rel="next" title="Dealing with Request Data" href="../request_data/index.html" />
    <link rel="prev" title="Important Terms" href="../terms/index.html" />
    <link rel="canonical" href="index.html">
  <script>DOCUMENTATION_OPTIONS.URL_ROOT = '../index.html';</script>
   
  
<!-- RTD Extra Head -->

<!-- 
Always link to the latest version, as canonical.
http://docs.readthedocs.org/en/latest/canonical.html
-->
<link rel="canonical" href="index.html" />

<link rel="stylesheet" href="../../../../assets.readthedocs.org/static/css/readthedocs-doc-embed.css" type="text/css" />

<script type="text/javascript" src="../_static/readthedocs-data.js"></script>

<!-- Add page-specific data, which must exist in the page js, not global -->
<script type="text/javascript">
READTHEDOCS_DATA['page'] = "unicode"
READTHEDOCS_DATA['source_suffix'] = ".rst"
</script>

<script type="text/javascript" src="../../../../assets.readthedocs.org/static/javascript/readthedocs-analytics.js" async="async"></script>

<!-- end RTD <extrahead> -->
</head><body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex/index.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="../py-modindex/index.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="../request_data/index.html" title="Dealing with Request Data"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="../terms/index.html" title="Important Terms"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="../index.html">Werkzeug Documentation (1.0.x)</a> &#187;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="unicode">
<span id="id1"></span><h1>Unicode<a class="headerlink" href="#unicode" title="Permalink to this headline">¶</a></h1>
<p>Since early Python 2 days unicode was part of all default Python builds.  It
allows developers to write applications that deal with non-ASCII characters
in a straightforward way.  But working with unicode requires a basic knowledge
about that matter, especially when working with libraries that do not support
it.</p>
<p>Werkzeug uses unicode internally everywhere text data is assumed, even if the
HTTP standard is not unicode aware as it.  Basically all incoming data is
decoded from the charset specified (per default <cite>utf-8</cite>) so that you don’t
operate on bytestrings any more.  Outgoing unicode data is then encoded into
the target charset again.</p>
<div class="section" id="unicode-in-python">
<h2>Unicode in Python<a class="headerlink" href="#unicode-in-python" title="Permalink to this headline">¶</a></h2>
<p>In Python 2 there are two basic string types: <cite>str</cite> and <cite>unicode</cite>.  <cite>str</cite> may
carry encoded unicode data but it’s always represented in bytes whereas the
<cite>unicode</cite> type does not contain bytes but charpoints.  What does this mean?
Imagine you have the German Umlaut <cite>ö</cite>.  In ASCII you cannot represent that
character, but in the <cite>latin-1</cite> and <cite>utf-8</cite> character sets you can represent
it, but they look differently when encoded:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="sa">u</span><span class="s1">&#39;ö&#39;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;latin1&#39;</span><span class="p">)</span>
<span class="go">&#39;\xf6&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="sa">u</span><span class="s1">&#39;ö&#39;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="go">&#39;\xc3\xb6&#39;</span>
</pre></div>
</div>
<p>So an <cite>ö</cite> might look totally different depending on the encoding which makes
it hard to work with it.  The solution is using the <cite>unicode</cite> type (as we did
above, note the <cite>u</cite> prefix before the string).  The unicode type does not
store the bytes for <cite>ö</cite> but the information, that this is a
<code class="docutils literal notranslate"><span class="pre">LATIN</span> <span class="pre">SMALL</span> <span class="pre">LETTER</span> <span class="pre">O</span> <span class="pre">WITH</span> <span class="pre">DIAERESIS</span></code>.</p>
<p>Doing <code class="docutils literal notranslate"><span class="pre">len(u'ö')</span></code> will always give us the expected “1” but <code class="docutils literal notranslate"><span class="pre">len('ö')</span></code>
might give different results depending on the encoding of <code class="docutils literal notranslate"><span class="pre">'ö'</span></code>.</p>
</div>
<div class="section" id="unicode-in-http">
<h2>Unicode in HTTP<a class="headerlink" href="#unicode-in-http" title="Permalink to this headline">¶</a></h2>
<p>The problem with unicode is that HTTP does not know what unicode is.  HTTP
is limited to bytes but this is not a big problem as Werkzeug decodes and
encodes for us automatically all incoming and outgoing data.  Basically what
this means is that data sent from the browser to the web application is per
default decoded from an utf-8 bytestring into a <cite>unicode</cite> string.  Data sent
from the application back to the browser that is not yet a bytestring is then
encoded back to utf-8.</p>
<p>Usually this “just works” and we don’t have to worry about it, but there are
situations where this behavior is problematic.  For example the Python 2 IO
layer is not unicode aware.  This means that whenever you work with data from
the file system you have to properly decode it.  The correct way to load
a text file from the file system looks like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">f</span> <span class="o">=</span> <span class="n">file</span><span class="p">(</span><span class="s1">&#39;/path/to/the_file.txt&#39;</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
    <span class="n">text</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>    <span class="c1"># assuming the file is utf-8 encoded</span>
<span class="k">finally</span><span class="p">:</span>
    <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>There is also the codecs module which provides an open function that decodes
automatically from the given encoding.</p>
</div>
<div class="section" id="error-handling">
<h2>Error Handling<a class="headerlink" href="#error-handling" title="Permalink to this headline">¶</a></h2>
<p>Functions that do internal encoding or decoding accept an <code class="docutils literal notranslate"><span class="pre">errors</span></code>
keyword argument that is passed to <code class="xref py py-meth docutils literal notranslate"><span class="pre">str.decode()</span></code> and
<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.encode" title="(in Python v3.8)"><code class="xref py py-meth docutils literal notranslate"><span class="pre">str.encode()</span></code></a>. The default is <code class="docutils literal notranslate"><span class="pre">'replace'</span></code> so that errors are easy
to spot. It might be useful to set it to <code class="docutils literal notranslate"><span class="pre">'strict'</span></code> in order to catch
the error and report the bad data to the client.</p>
</div>
<div class="section" id="request-and-response-objects">
<h2>Request and Response Objects<a class="headerlink" href="#request-and-response-objects" title="Permalink to this headline">¶</a></h2>
<p>As request and response objects usually are the central entities of Werkzeug
powered applications you can change the default encoding Werkzeug operates on
by subclassing these two classes.  For example you can easily set the
application to utf-7 and strict error handling:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">werkzeug.wrappers</span> <span class="k">import</span> <span class="n">BaseRequest</span><span class="p">,</span> <span class="n">BaseResponse</span>

<span class="k">class</span> <span class="nc">Request</span><span class="p">(</span><span class="n">BaseRequest</span><span class="p">):</span>
    <span class="n">charset</span> <span class="o">=</span> <span class="s1">&#39;utf-7&#39;</span>
    <span class="n">encoding_errors</span> <span class="o">=</span> <span class="s1">&#39;strict&#39;</span>

<span class="k">class</span> <span class="nc">Response</span><span class="p">(</span><span class="n">BaseResponse</span><span class="p">):</span>
    <span class="n">charset</span> <span class="o">=</span> <span class="s1">&#39;utf-7&#39;</span>
</pre></div>
</div>
<p>Keep in mind that the error handling is only customizable for all decoding
but not encoding.  If Werkzeug encounters an encoding error it will raise a
<a class="reference external" href="https://docs.python.org/3/library/exceptions.html#UnicodeEncodeError" title="(in Python v3.8)"><code class="xref py py-exc docutils literal notranslate"><span class="pre">UnicodeEncodeError</span></code></a>.  It’s your responsibility to not create data that is
not present in the target charset (a non issue with all unicode encodings
such as utf-8).</p>
</div>
<div class="section" id="the-filesystem">
<span id="filesystem-encoding"></span><h2>The Filesystem<a class="headerlink" href="#the-filesystem" title="Permalink to this headline">¶</a></h2>
<details class="changelog">
<summary>Changelog</summary><div class="versionchanged">
<p><span class="versionmodified">Changed in version 0.11.</span></p>
</div>
</details><p>Up until version 0.11, Werkzeug used Python’s stdlib functionality to detect
the filesystem encoding. However, several bug reports against Werkzeug have
shown that the value of <a class="reference external" href="https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding" title="(in Python v3.8)"><code class="xref py py-func docutils literal notranslate"><span class="pre">sys.getfilesystemencoding()</span></code></a> cannot be
trusted under traditional UNIX systems. The usual problems come from
misconfigured systems, where <code class="docutils literal notranslate"><span class="pre">LANG</span></code> and similar environment variables are not
set. In such cases, Python would default to ASCII as filesystem encoding, a
very conservative default that is usually wrong and causes more problems than
it avoids.</p>
<p>Therefore Werkzeug will force the filesystem encoding to <code class="docutils literal notranslate"><span class="pre">UTF-8</span></code> and issue a
warning whenever it detects that it is running under BSD or Linux, and
<a class="reference external" href="https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding" title="(in Python v3.8)"><code class="xref py py-func docutils literal notranslate"><span class="pre">sys.getfilesystemencoding()</span></code></a> is returning an ASCII encoding.</p>
<p>See also <a class="reference internal" href="../filesystem/index.html#module-werkzeug.filesystem" title="werkzeug.filesystem"><code class="xref py py-mod docutils literal notranslate"><span class="pre">werkzeug.filesystem</span></code></a>.</p>
</div>
</div>


          </div>
        </div>
      </div>
  <span id="sidebar-top"></span>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  
    
            <p class="logo"><a href="../index.html">
              <img class="logo" src="../_static/werkzeug.png" alt="Logo"/>
            </a></p>
  

  <h3>Contents</h3>
  <ul>
<li><a class="reference internal" href="#">Unicode</a><ul>
<li><a class="reference internal" href="#unicode-in-python">Unicode in Python</a></li>
<li><a class="reference internal" href="#unicode-in-http">Unicode in HTTP</a></li>
<li><a class="reference internal" href="#error-handling">Error Handling</a></li>
<li><a class="reference internal" href="#request-and-response-objects">Request and Response Objects</a></li>
<li><a class="reference internal" href="#the-filesystem">The Filesystem</a></li>
</ul>
</li>
</ul>
<h3>Navigation</h3>
<ul>
  <li><a href="../index.html">Overview</a>
    <ul>
          <li>Previous: <a href="../terms/index.html" title="previous chapter">Important Terms</a>
          <li>Next: <a href="../request_data/index.html" title="next chapter">Dealing with Request Data</a>
    </ul>
  </li>
</ul>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <div class="searchformwrapper">
    <form class="search" action="https://werkzeug.palletsprojects.com/en/1.0.x/search/" method="get">
      <input type="text" name="q" />
      <input type="submit" value="Go" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    </div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
  
    <div class="footer" role="contentinfo">
        &#169; Copyright 2007 Pallets.
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.8.5.
    </div>
    <script>
      if (typeof READTHEDOCS_DATA !== 'undefined') {
        if (!READTHEDOCS_DATA.features) {
          READTHEDOCS_DATA.features = {};
        }
        READTHEDOCS_DATA.features.docsearch_disabled = true;
      }
    </script>

  </body>

<!-- Mirrored from werkzeug.palletsprojects.com/en/1.0.x/unicode/ by HTTrack Website Copier/3.x [XR&CO'2014], Tue, 15 Sep 2020 06:37:09 GMT -->
</html>